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1. Applicant herewith submits to the United States Designated Office (DO/US) the following items 
under35U.S.C.371: 

a. [x] This express request to immediately begin national examination procedures (35 
U.S.C. 371(f)). 

b. [x] The U.S. National Fee (35 U.S.C. 371(c)(1)) and other fees (37 C.F.R. 1.492), as 
indicated below: 



2. Fees 



CLAIMS 
FEE 


(1) FOR 


(2) NUMBER 
FILED 


(3) NUMBER 
EXTRA 


(4) RATE 


(5) CALCULATIONS 


A 


TOTAL 
CLAIMS 


84 - 20 = 


64 


x$ 18.00= 


$ 1,152.00 




INDEPENDEN 
T 


9-3 = 


6 


x$ 78.00= 


468.00 




CLAIMS 












MULTIPLE DEPENDENT CLAIMS(S) (if applicable) + $260.00 




FEE** 


The international search fee, as set forth in § 1.445(a)(2) to 
be paid to the US PTO acting as an international 
Searching Authority: 






[ ] has been paid (37 CFR 

1.492(a)(2)) $760.00 

[f/^] has not been paid (37 CFR 
1.492(a)(3)) $970.00 








[ ] where a search report on the international 

application has been prepared by the European 
Patent Office or the Japanese Patent Office (37 
CFR 1.492(a)(5)) $840.00 






Total of above Calculations 


=970 


SMALL 
ENTITY 


Reduction by for filing by small entity, if applicable. Affidavit 
must be filed also, (note 37 CFR 1.9, 1.27, 1.28) 






Subtotal 






Total National Fee 


$970.00 




Fee for recording the enclosed assignment document $40.00 (37 
CFR 1.21(h)). (See Item 10 below). See attached "ASSIGNMENT 
COVER SHEET (37 CFR 3.34)". 




TOTAL 


Total Fees enclosed 


$970.00 



"^See attached Preliminary Amendment Reducing the Number of Claims, 

**WARNING: "To avoid abandonment of the application, the applicant shall furnish to the United States Patent and 
Trademark Office not later than the expiration of 20 months from the priority date; *** (2) the basic 
national fee (see § 1, 492(a)). The 20-month time limit may not be extended " 37 C.F.R § L 494(b). 
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] A check in the amount of $ 970.00 to cover the above fees is enclosed. 

[ ] Please charge Account No. in the amount of $ . 



A duplicate copy of this sheet is enclosed. 



If the translations of the international application, oath or declaration and national fee have not been 
submitted by the applicant within twenty (20) months from the priority date, the applicant will be so 
notified and given a period of time within which to file the translation and/or oath or declaration in 
order to prevent abandonment The payment of the surcharge set forth in § 1. 492(e) is required as a 
condition for accepting the oath or declaration later than twenty (20) months after the priority date. The 
payment of the processing fee set forth in § L492(f) is required for acceptance of an English translation 
later than twenty (20) months after the priority date. Failure to comply with these requirements will 
result in abandonment of the application. The provisions of§LlS6 will apply. 37 CAR § 1.494(c); 
Notice of Jan. 7, 1993, 1147 O.G. 29 to 40, at 35. 

of the International application as filed (35 U.S.C. 371(c)(2)): 
[ ] is transmitted herewith. 

[ ] is not required, as the application was filed with the United States Receiving 
Office. 

has been transmitted 

i. [✓] by the International Bureau. Date of mailing of the application fi*om form 

PCT/IB/308): . 

ii- [ ] by applicant on . 

Date 

NOTE: Section 1.494(b) was amended to require that the basic national fee and a copy of the international application 
must befded with the Office by 20 months from the priority date to avoid abandonment. "The International 
Bureau nominally provides the copy of the international application to the Office in accordance with PCT Article 
20. At the same time, the International Bureau notifies the applicant of the communication to the Office, In 
accordance with PCT Rule 47. h that notice shall be accepted by all designated offices as conclusive evidence that 
the communication has duly taken place. Thus, if the applicant desires to enter the national stage and applicant 
has received notice from the International Bureau, applicant need only pay the basic national fee by 20 months 
from the priority date. " [This can now be paid subsequently with a surcharge.] Notice of Jan. 7, 1993, 1147 O.G. 
29 to 40, at 35. 

4. A translation of the International application into the English language (35 U.S.C. 371(c)(2)): 

a. [*/] is transmitted herewith. 

b. [ ] is not required as the application was filed in English. 

c. [ ] was previously transmitted by applicant on . 

Date 



1 

ii. 



WARNING: 



A copy 
a. 
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5 . [/] Amendments to the claims of the International application under PCX Article 1 9 (3 5 
U.S.C. 371(c)(3)): 

NOTE: The Notice of January 7, 1993 indicates that 37 C.F.R. § L494(d) was "amended to clarify the existing practice 
that PCT Article 19 Amendments must be submitted by 20 months from the priority date, which time may not be 
extended. " This Notice further advises: "Of course, the failure to do so does not result in loss of the subject matter 
of PCT Article 19 amendments. The applicant may submit that subject matter in a preliminary amendment filed 
under Section 1. 121. In many cases, filing an amendment under Section L 121 is preferable since grammatical or 
idiomatic errors may be corrected" 1147 0,G. 29-40, at 35. See item 11(c) below, 

a. [ ] are transmitted herewith. 

b. [ ] have been transmitted 

i. [ ] by the International Bureau. Date of mailing of the amendment (from form 
PCT/IB/308): . 

ii. [ ] by applicant on . 

Date 

c. [%/ ] have not been transmitted, as 

i. [ ] no notification has been received that the International Search Authority 

has received the Search Copy, 
ii [ ] the Search Copy was received by the International Searching Authority, 

but the Search Report has not yet been issued. Date of receipt of Search 

Copy prom form PCT/ISA/202): . 

iii. [/] applicant chose not to make amendments under PCT Article 19, Date of 
mailing of Search Report (from form PCT/ISA/210): 

iv. [ ] the time limit for the submission of amendments has not yet expired. The 
amendments, or a statement that amendments have not been made, will be 
transmitted before the expiration of the time limit under PCT Rule 46.1. 

6. [,/^] A translation of the amendments to the claims under PCT Article 19 (35 U.S C 
371(c)(3)): 

a. [ ] is transmitted herewith. 

b. [ ] is not required as the amendments were made in the English language. 

c. [/] has not been transmitted for reasons indicated at point 5(c) above. 

7. [✓] An oath or declaration of the inventor (35 U.S.C. 371(c)(4)) complying with 35 U.S.C. 
115 

a. [ ] was previously submitted by applicant on 

Date 

b. is submitted herewith, and such oath or declaration 

i. [ ] is attached to the application. 

ii. [/] identifies the application and any amendments under PCT Article 19 that 
were transmitted as stated in points 3(b) or (c) and 5(b); and states that they 
were reviewed by the inventor, as required by 37 C.F.R. 1.70. 

iii. [ ] will follow. 
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11. Other document(s) or information included: 

8. [/] An international Search Report or Declaration under PCT Article 17(2)(a): 

a. [ ] is transmitted herewith. 

b. [ ] has been transmitted by the International Bureau. Date of mailing from form 
PCT/IB/308): . 

c. [ ] is not required, as the application was searched by the United States International 
Searching Authority. 

d. [/] will be transmitted promptly upon request. 

e. [ ] has been submitted by applicant on . 

Date 

f. [ ] is not transmitted, as the international search has not yet issued. 
9. [✓] An information Disclosure Statement under 37 C.F.R. 1.97 and 1.98: 

a. [ ] is transmitted herewith. 

Also transmitted herewith is (are) 
[ ] Form PTO-1449 (PTO/SB/08A and 08B) 
[ ] Copies of citations listed 

b. [✓] will be transmitted within THREE MONTHS of the date of submission of 
requirements under 35 U.S.C. 371(c). 

c. [ ] was previously submitted by applicant on . 

Date 

10. [/] An assignment document is transmitted herewith for recording. A separate 

[✓] "COVER SHEET FOR ASSIGNMENT (DOCUMENT) ACCOMPANYING NEW 
PATENT APPLICATION" or 

[]FORM PTO— 1595 
is also attached. 

[y] Please mail the recorded assignment document to: 

i. [✓] the person whose signature and address appears below. 

ii. [ ] the following: 

1 1 . [y^] Additional documents 

a. [✓] Copy of request (PCT/RO/1 0 1 ) 

b. [ ] International Publication No. 

i. [ ] Specification, claims and drawing 

ii. [ ] Front page only 

c. [/] Preliminary amendment (37 C.F.R. § 1.121) 

d. [^] Other POWER OF ATTORNEY BY GURMINDER SINGH 

POWER OF ATTORNEY BY VINOD VIJAYALEKSHMI 
VASUDEVAN 

12. [%^] The above checked items are being transmitted 

a. [ ] before the 1 8th month publication. 

b. [✓] after publication and the article 20 communication, but before 20 months from 
the priority date. 

c. [ ] after 20 months (revival). 

NOTE: Petition to revive (37 C.F.K }J37(a) or (b)) is necessary if 35 U.S.C, 371 requirements are submitted after 20 
months. 
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13. [ ] Certain requirements under 35 U.S.C. 371 were previously submitted by the 

applicant on namely: 

Date 



AUTHORIZATION TO CHARGE ADDITIONAL FEES 

WARNING: Accurately count claims, especially multiple dependent claims, to avoid unexpected high charges if extra 
claims are authorized. 

NOTE: "'A written request may be submitted in an application that is an authorization to treat any concurrent 
or future reply, requiring a petition for an extension of time under this paragraph for its timely 
submission, as incorporating a petition for extension of time for the appropriate length of time. An 
authorization to charge all required fees, fees under § LI 7, or all required extension of time fees will 
be treated as a constructive petition for an extension of time in any concurrent or future reply 
requiring a petition for an extension of time under this paragraph for its timely submission. 
Submission of the fee set forth in § 1.17 (a) will also be treated as a constructive petition for an 
extension of time in any concurrent reply requiring a petition for an extension of time under this 
paragraph for its timely submission. "37 CFR LI 36(a)(3). 

NOTE: ''Amounts of twenty-five dollars or less will not be returned unless specifically requested 
within a reasonable time, nor will the payer be notified of such amounts; amounts over 
twenty-five dollars may be returned by check or, if requested, by credit to a deposit account. " 
37CFRL26(a). 

[/] The Commissioner is hereby authorized to charge the following additional fees that 
may be required by this paper and during the entire pendency of this application to 
Account No. 12-0425 



M 37 C.F.R. 1 .492(a)(lX (2), (3), and (4) (filing fees) 

WARNING: Because failure to pay the national fee within 20 months without extension (37 

C.F.R § L494(b)(2)), results in abandonment of the application, it would be best to 
always check the above box. 

[ ] 37 C.F.R. 1 .492(b), (c), and (d) (presentation of extra claims) 

NOTE: Because additional fees for excess or multiple dependent claims not paid on filing or on later presentation must 

only be paid or these claims cancelled by amendment, prior to the expiration of the time period set for response by 
the PTO in any notice of fee deficiency (37 C.F.R, 1.16(d)), it might be best not to authorize the PTO to charge 
additional claim fees, except possibly when dealing with amendments after final action. 

[*/] 37 C.F.R. 1.17 (application processing fees) 

[] 37 CFR 1.17(a)(l)-(5)(extension fees pursuant to § 1.136(a). 

[ ] 37 C.F.R. 1.18 (issue fee at or before mailing of Notice of Allowance, 
pursuant to 37 C.F.R. 1 .3 1 1(b)). 

NOTE: Where an authorization to charge the issue fee to a deposit account has been filed before the 
mailing of a Notice of Allowance, the issue fee will be automatically charged to the deposit 
account at the time of mailing the notice of allowance, 37 C.F.R. 1.311(b). 
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NOTE: 37 C.F.R. 1.28(b) requires ^^Notification of any change in status resulting in loss of 

entitlement to small entity status must be filed in the application . . . prior to paying or at the 
time of paying . . . issue fee,... " From the wording of 37 C.F.K L 28(b): (a) notification of 
change of status must be made even if the fee is paid as '^other than a small entity" and (b) no 
notification is required if the change is to another small entity. 



37 C.F.R. 1.492(e) and (f) (surcharge fees for filing the declaration and/or 
filing an English translation of an Intemationg^^lication later than 20 
months after the priority date. /I / 
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PATENT 

IN THE UNITED STATES PATENT AND TRADEMARK OFFICE 

In re application of: GURMINDER SINGH, et al 

For: METHOD AND APPARATUS FOR CONTENT-LINKING SUPPLEMENTAL 
INFORMATION WITH TIME-SEQUENCE DATA 



Attorney Docket No.: U 012304-4 



Assistant Commissioner for Patents 
Washington, D.C. 20231 

Sir: 

PRELIMINARY AMENDMENT 

Please amend the above application as follows: 
IN THE CLAIMS 

Claim 19, line 1, delete "any one of claims 16 to 18" and replace therefor 

~ claim 16— 



Claim 33, lines 1-2, delete "any one of claims 30 to 32" and replace therefor 

~ claim 30— 



CERTIFICATE UNDER 37 1.10 

I hereby certify that this paper is being deposited with the United States Postal Service on this 
date JULY 8. 1999 in an envelope as "EXPRESS MAIL POST OFFICE TO ADDRESSEE" 
Mailing Label Number EE784099672US addressed to the: Commissioner of Patents and 
Trademarks, Washington, D.C. 20231 

JENNIFER RASHKIN 

(Type or print name of person mailing paper) 

^Sffiiature of pelson mailing paper) ^ 
NOTE: Each paper or fee referred to as encl^ed herein has the number of the "EXPRESS 
MAIL" mailing label place thereon prior to mailing 37 CFR 1.16(b). 
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Claim 47, line 1, delete "or 46" 



Claim 57, line 1, delete "or 56" 



Claim 67, line 1, delete "or 66" 
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FIELD OF THE INVENTION 

5 The present invention is directed to the field of delivering animation, audio and video 
data via broadcasting or network mechanisms, and in particular to systems for 
providing value-added content to animation, audio and video programming. 

BACKGROUND 

1 0 wiikiSm rapid ad^^^nt of technologies such as digital multimedia, broadband 

networking, video on demand, and the Internet, large volumes of audio, animation and 
video data or programming are readily available to the public. This programming can 
be deUvered to an audience by a multitude of ways including broadcasting, on-demand 
type systems, etc. However, there are not adequate mechanisms to distinguish 
1 5 programming distributed in this fashion fi*om traditional forms such as broadcast 
television or screened movies. 

There are several known systems which offer video-on-demand and pay per-view 
services. These systems afford an opportunity for serving the audience with additional 
20 information about the programming. Existing mechanisms require the user to actively 
request any additional information. However, these mechanisms have a number of 
drawbacks. A first disadvantage is that a user must have adequate knowledge of their 
own to request any additional information. The information is not associated with the 
content of the programming. Still fiirther, there is no guarantee that the user will 
25 request and view the additional information that is available. This is a major 
impediment, particularly in advertising. 

In existing methods for normal TV broadcast situations to systems such as 
Synchronised Multimedia Integration Language (SMIL), Microsoft Netshow, MPEG- 
30 4, etc., the programming is required to be created, or packaged, afi*esh along with the 
associated information into a single entity. This has a number of disadvantages 
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including a lack of flexibility and programmable control. Further, it cannot be done on 
the fly. If the associated information changes, the entire package must be redone. 
That is, additional information cannot be changed with online digital content or 
broadcast material in current formats, unless there is a repackaging and/or format 
5 conversions. 

Thus, a significant disadvantage of such conventional systems is that they do not 
permit a service provider, who offers audio, animation and video data services, to 
associate additional information with the programming and provide the two without 
10 interfering with the programming production process. Existing systems also fail to 

provide additional information with the digital media content without changing the way 
in which the content is produced, stored, and/or distributed. 

SUMMARY 

15 The following aspects of the invention are directed to ameliorating or overcoming one 
or more disadvantages of conventional systems including those described above. 

In accordance with a first aspect of the invention, a method is provided for presenting 
supplemental information and/or action with time-sequence data. The time-sequence 
20 data can be animation, audio, and/or video data. The method includes the steps of 
providing time-sequence data; separately providing supplemental information, action, 
or both, fi-om the time-sequence data; and linking the supplemental information, action, 
or both, with the time-sequence data during playback of the time-sequence data. 

25 Preferably, the supplemental information and/or action is linked with the time-sequence 
data by a start time and a stop time, or a start time and a duration period. 
Alternatively, the supplemental information and/or action can be linked with the time- 
sequence data by annotation information associated with the time-sequence data. Yet 
ftirther, the supplemental information and/or action can be linked -v^ith the time- 

30 sequence data using on-the-fl>' content analysis of the time-sequence data during 
playback of the latter. 
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Preferably, the supplemental information and/or action is content-linked with the time 
sequence data. The time-sequence data can be provided by broadcast or via a 
network. The supplemental information and/or action is preferably provided remotely 
5 via a network. It can include a displayed prompt with which a user can interact by user 
input and a link to a remote location. The link is more preferably a universal resource 
locator (URL). The following mechanisms can also be provided remotely via a 
network. 

10 Preferably, the time-sequence data and the supplemental information and/or action are 
linked by a Unk mechanism. Two or more of the link mechanism, the time-sequence 
data, and the supplemental mformation and/or action are initially co-located at their 
source. Alternatively, two or more of the link means, the time-sequence data, and the 
supplemental information and/or action are initially remotely located at one or more 

1 5 source locations. 

In accordance with another aspect of the invention, an apparatus is provided for 
presenting supplemental information and/or action with time-sequence data. The 
apparatus includes: a device for providing time-sequence data; a device for separately 
20 providing supplemental information and/or action from the time-sequence data; and a 
device for linking the supplemental information and/or action with the time-sequence 
data during playback of the time-sequence data. 

In accordance with yet another aspect of the invention, a computer program product is 
25 provided having a computer readable medium having a computer program recorded 
therein for presenting supplemental information and/or action with time-sequence data. 
The computer program includes: a module for providing time-sequence data; a module 
for separately providing supplemental information and/or action from the time- 
sequence data; and a module for linking the supplemental information and/or action 
30 with the time-sequence data during playback of the time-sequence data 
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BRIEF DESCRIPTION OF THE DRAWINGS 

Embodiments of the invention are described hereinafter with reference to the drawings, 
in which: 

5 Fig. 1 A is a timing diagram graphically depicting a linked combination of time- 
sequence data and supplemental information and/or action in accordance with 
embodiments of the invention; 

Fig. IB is a timing diagram illustrating the linked relationship of exemplary video, 
1 0 animation and/or audio data and four advertisements in accordance with embodiments 
of the invention; 

Fig. 2 is a block diagram illustrating the functional modules of an apparatus for 
presenting supplemental information, action, or both, with time-sequence data in 
1 5 accordance with the a embodiment; 

Fig. 3 is a block diagram illustrating a variation of the apparatus according to the first 
embodiment shown in Fig. 2; 

20 Fig. 4 is a graphical depiction of an exemplary application of the embodiments where a 
video is displayed on a monitor with a corresponding advertising banner; 

Fig. 5 is a high-level flow diagram of a method of delivering supplemental information 
and/or action with time-sequence data in accordance with the first embodiment of the 
25 invention; 

Fig. 6 is a flow diagram of a method of providing value-added content related to at 
least one of animation, audio and video data, in the process of presenting the 
animation, audio, and/or video data to a user in accordance with a second embodiment 
30 of the invention: 
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Fig. 7 is a flow diagram of a method of distributing advertisements with animation, 
audio and/or video data delivered for presentation by way of broadcast or streaming 
over a network in accordance with a third embodiment of the invention; and 

5 Figs. 8A-8E are detailed flow diagrams of a method of presenting advertisements in 
association with video or movie in accordance with a fourth embodiment of the 
invention. 

DETAILED DESCRIPTION 

1 0 A method, apparatus and computer program product for Unking supplementary 

information, actions, or both, to time-sequence (or "time-base) data is disclosed. The 
time-sequence data include animation, audio and/or video data. Fig. 1 A is a timing 
diagram illustrating a Unked combination 100 of time-sequence data 110 and 
supplemental information and/or action 120, 122, 124. Three portions 120, 122, 124 

1 5 are linked for presentation with the time-sequence data. Essentially numerous forms of 
supplemental mformation and/or actions can be linked to the time-sequence data using 
the embodiments of the invention described herein. In addition to providing 
supplemental information, the embodiments can also provide "actions", which in broad 
terms denotes any additional functionality provided to the user. This can be 

20 implemented as a computer program such as a Java applet that can be executed by a 
computer displaying time-sequence data. Exemplary actions include functionality for 
conducting surveys or for changing time-sequence data. 

Amongst other things, the embodiments enable video service providers such as cable 
25 and broadcast television service providers to add controUably and on-the-fly added- 
value content to their video product by tightly linking advertisements to the subject- 
matter content of the video product. In this manner, the video with content-linked 
supplemental information and/or actions can be readily differentiated from normal 
broadcast video, for example. The same applies equally to audio and animation data. 
30 The supplemental information and/or actions are provided via a network, preferabl}' an 
electronic network such as the Internet or an Intranet. 
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Fig IB is a timing diagram illustrating the linked relationship 150 between a video, 
animation and/or audio program 160 and four advertisements 162, 164, 166, 168. 
Additional advertisement content may be available at a remote location on a network 

5 and be accessed by a link (e.g. URL) in the advertisement. As shown, the estimation 
150 includes overlapping advertisements, namely first and third advertisements 162 and 
166 on the one hand and the second advertisement 164 on the other. One or more of 
the video, animation and/or audio data, the supplemental information and/or action, 
and linking mechanisms or controls for linking the content data with the supplemental 

10 information and/or action can be distributed via a network. Further, they may be 
obtained fi-om two or more different sources on the network. Thus, for example, a 
video content provider can provide the video programming via the Internet from a first 
source site, the supplemental information and/or action fi-om another source site, and 
the linking control database fi-om yet another site. Other combinations are possible. 

15 For example, the video programming may be available locally at a user's location while 
the other components are delivered via a network, or vice versa. 

With reference to the example of video delivered by broadcast or a network, the 
embodiments of the invention enable advertisments to be tightly linked in a 

20 programmable and on-the-fly manner to the content of the video. An advertisement 
can be displayed as a graphical banner in combination with the video presented on a 
display monitor. Fig. 4 is a depiction of an exemplary application 400 of the 
embodiments where a video 420 is displayed with a corresponding advertising banner 
430. In this example, the video data 420 is displayed in a panel of the graphical 

25 operating system 410 (e.g. Microsoft Windows 95 or 98). The exemplary portion of 
the video 420 is currently displaying a well-known water fountain at Sentosa Island in 
Singapore. In conjunction with the video 420 being presented, an advertisment 430 is 
presented within the same panel It will be readily appreciated that the advertisment 
430 need not be presented in the same window^ The advertisment 430 is presented as 

30 a banner referring to "Destination Sentosa: Experience it all. . A user can click on 
the banner 430 while it is displayed, and further supplemental information and/or 
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action can be obtained. For example, an advertisement 440 about Sentosa Island 
(shown in the background) can be retrieved via a URL of the banner 430 to dovwiload 
the advertisement 440 from a site on the Internet. 

5 If the video is a fihn with actors and props, the advertisements can be directly linked to 
such video content. An advertiser can make direct reference in the advertisement to 
props in the video including articles of clothing, jeweky and accessories worn by 
particular actors in a scene. The advertiser can also make reference to items such as 
furniture, wallpaper and appliances. The foregoing applies to virtually anything visibly 

10 and/or audibly perceptible in the video. Thus, the advertisement content-linked to the 
video can bring an item in the video to the viewer's attention and provide a way of 
making a purchase by the viewer using electronic commerce, for example. 

Again, the video can be streamed over an electronic network to a user's personal 
1 5 computer with video display capabilities. By way of example, the video itself can be 
displayed in a window in a graphical operating system such as Microsoft Windows 95 
(Trade Mark) running on a conventional personal computer, laptop computer, or work 
station. Also displayed in the window can be a banner or other graphical presentation 
containing information associating the banner with the content of the video. 
20 Alternatively, the banner might be displayed in a separate window. By means of a 
textual reference, the banner can refer to a dress worn by an actress in the current 
video content as being sold by a particular advertiser. By appropriate interaction with 
the banner, the user can obtain fiirther information about the dress or purchase it. The 
information obtained in this manner can be obtained remotely or in appropriate cases 
25 locally. In one embodiment, the banner displays an HTML link to a remote site to 

which the user's personal computer is connectable by means of the Internet. From the 
advertiser's remote site, further information about the dress can be obtained including 
how to purchase it using electronic commerce or by telephone purchase. 

30 In yet another illustration, a video track can be played on a display device such as a 
television receiver with touch-screen functionality. In conjunction with the television 
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receiver, a set topbox (associated -with cable television services, for example) can be 
provided by v^rhich supplemental information and/or actions can be distributed. The 
functionality of such cable set topboxes are increasingly more sophisticated. An action 
can be displayed via the set topbox with the video on the television receiver, whereby 

5 the relevant action can be chosen from a banner or the Uke displayed with the video. 
For example, the subject matter content of the displayed video can be a documentary 
about bears and include material on bears in Alaska. Where a segment about Alaskan 
bears is displayed, a banner displayed in association with the video can show a message 
indicating that a related program about Alaska is currently available for viewing. By 

1 0 appropriate user interaction with the banner, such as touching an icon displayed in the 
banner, an action is carried out causing the displayed programming to change to the 
related program about Alaska without the user having to manually select the particular 
station. 

15 In still another illustration, an audio track received by a tuner/receiver provided in a 
personal computer (e.g. Web radio) can be audibly output by the personal computer. 
In conjunction with the content of the audio segment, supplemental information and/or 
action can be displayed on the personal computer. The subject matter of the audio 
segment might involve discussion about some form of on-going audience survey, and 

20 an appropriate prompt can be displayed on the personal computer dependent upon tiie 
audio content. By clicking on the prompt, tiie user can interactively participate in tiie 
survey over the Internet. Numerous otiier appUcations can be envisaged by those 
skilled in the art in view of the disclosure of the embodiments herein. 

25 The embodiments of the invention are able to strongly link supplemental information 
and/or actions including advertising with the content of time-sequence data including 
animation, audio and video. Further, the embodiments of the invention enable flexible 
control of the flow of the supplemental information and/or action and its content. For 
example, the embodiments of the invention enable flexible control of the flow of 

30 advertising in relation to an animation, audio and/or video sequence and the content of 
the advertising itself Still furtiier. the embodiments of tiie invention in particular 
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appUcations enable close linkage of electronic commerce with the content of the time- 
sequence data. 

The embodiments of the invention are able to differentiate time-sequence data from 
conventional broadcast time-sequence data by providing a content-based linkage with 
the supplemental information, action or both. Further, the embodiments provide a 
non-interfering mechanism for providing the supplemental information, action or both 
with respect to both the time-sequence data itself and the deUvery mechanism used to 
deliver the time-sequence data. That is, no modification of the time-sequence data or 
its delivery mechanism is required. All existing time-sequence data can be used. For 
example, a video delivered on demand over a network to an end-user does not require 
any modification and can be sourced from a digital video disc (DVD) without change 
or embedding additional information prior to transmission. Still fiirther, the 
embodiments support customised and focused content-linking of supplemental 
information, action, or both to the time-sequence data. This is particularly 
advantageous where the supplemental information, action or both relates to 
advertising. In addition, the embodiments of the invention enable fiall, on-the-fly or 
"realtime" control over the content, sequencing and timing of the supplemental 
information, action, or both, delivered to the user in conjunction with the time- 
sequence data. Thus, advertisers for example can have fiill and realtime control over 
the advertisement content. Customised advertising can therefore be supported. 

As an example, embodiments of the invention permit a video program to be delivered 
via a network on demand or on a pre-scheduled basis to viewers. Sponsors of the 
25 program or advertisers can buy slots, both in terms of time-length as well as the time 
position in the video program. During the time slots, the sponsor or advertiser can 
pro\dde HTML, XML, or other format advertisements that appear on the viewer's 
display device (e.g. a personal computer). The sponsor or advertiser can have fiiU 
control over their advertisement content, whereas the advertisement length and 
30 position can be pre-negotiated and controlled by the deliver}^ service provide (e.g. 
cable servdce provider). The sponsor or advertiser provides a URL from where their 
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advertisement content can be retrieved or downloaded. An advertisement delivered in 
this manner can provide a link to the sponsor's or advertiser's electronic commerce 
facility via a network or provide other means of conducting business. The latter might 
include providing a phone number to call for telephone transactions or the like. 
5 Alternatively, a call centre can be contacted using video, audio or text communication. 

Fig. 5 is a high-level flow diagram illustrating a method of delivering supplemental 
information, action, or both, with time-sequence data in accordance with a first 
embodiment of the invention. Processing commences in step 500. In step 502, time 

1 0 sequence data is provided. Again, the time-sequence data can include one or more of 
animation, audio and video data or programming. The time-sequence data can be 
provided by broadcast or via a network. In step 504, supplemental information and/or 
action is provided separately from the time-sequence data. Examples of the 
supplemental information and/or action include a universal resource locator (URL) to a 

15 remote site via a network, an interactive program, an icon, or an advertisement banner. 
In step 506, the supplemental mformation and/or action is linked with the time- 
sequence data during or immediately before playback of the time-sequence data. The 
supplemental information and/or action is preferably provided remotely via a network. 
The supplemental information and/or action can include a link to a remote location, 

20 and in particular, a universal resource locator (URL). It can also include a displayed 
prompt with which user can interact by user input. 

The supplemental information and/or action can be Imked with the time-sequence data 
by means of a start time and a stop time, or a start time and a specified duration. 

25 Alternatively, the supplemental information and/or action can be linked with the time- 
sequence data by means of annotation information associated with the time-sequence 
data. Still fiirther, the supplemental information and/or action can be linked with the 
time-sequence data, which is dependent upon on-the-fly content analysis of the time- 
sequence data during playback of the latter. This may be done using any of a number 

30 of known image/video/audio segmentation, recognition and analysis techniques well 
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known to those skilled in the art. Preferably, the supplemental information and/or 
action is content-linked with the time sequence data. 



In step 508, the time-sequence data and the supplemental information and/or action is 
5 presented to an audience. In the case of video and animation information, presentation 
can be implemented using a display monitor of a personal computer, a television 
receiver, a video monitor or the like. In the case of audio information, the audio data 
(i.e. time sequence data) can be output by a personal computer (e.g. Web radio) or 
other conventional soimd reproduction devices for broadcast and network streamed 
10 audio. The supplemental information and/or action can be presented on a personal 
computer, a video monitor, or the Uke. 

The time-sequence data and the supplemental information and/or action are linked by a 
link. At least two of the link, the time-sequence data, and the supplemental 
15 information and/or action can be initially co-located at their source. Alternatively, two 
or more of the links, the tnne-sequence data, and the supplemental mformation and/or 
action can be initially remotely located at one or more remote source locations of the 
network. In step 510, processing terminates. 

20 Preferably, the first embodiment of the invention depicted in Fig. 5 is unplemented as 
software or a computer program for presenting supplemental information and/or action 
with time-sequence data. The software including a Java applet can be ran on a 
computer such as a personal computer or workstation, or Java virtual machine, for 
example, well knovm to those skilled in the art. In particular, the computer program 

25 can be recorded on a computer readable mediiim and used in combination v^th the 

computer as a computer program product. The steps of Fig. 5 can be implemented as 
modules of software containing operations or instructions to implement the 
ftmctionality. Alternatively, the fimctionality of Fig. 5 can be implemented 
electronically (e.g. in an application specific integrated circuit ASIC). For example, it 

30 can be embedded in a set top box for use udth a video monitor or television receiver. 
Numerous other applications will be readily appreciated by those skilled in the an in 
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view of the disclosure herein. The foregoing applies equally to further embodiments of 
the invention described hereinafter. 



Fig. 2 is a block diagram illustrating the functional modules of an apparatus 200 for 
5 presenting supplemental information, action, or both, with time-sequence data in 
accordance with the first embodiment. Again, time-sequence data including video, 
animation and/or audio data 210 are provided by broadcast or via an electronic 
network. Alternatively, the data 210 may be provided locally. In addition, one or more 
linking mechanisms 220 is provided separately from the time-sequence data 210. Two 
10 or more linkmg mechanisms 220 are referred to in Fig. 2 as a database (DB) of linking 
mechanisms. The linking mechanism database can be prepared by video, animation, 
and/or audio content provider or the operator. Supplemental information and/or 
action (or value added content) 230 is also provided. This can be HTML/XML/other 
formats advertisements prepared and controlled by advertisers. Preferably, a linking 
15 mechanism 220 is a predefined data structure containing parameters of particular 

supplemental information and/or action 230 for presentation with a particular portion 
of the time-base data 210. Preferably, the database 220 is database of advertisement 
controls, and the supplemental mformation and/or action 230 is one or more 
advertisements. 

20 

One or more of the video, animation and/or audio data 210, the supplemental 
information and/or action 230 and linking mechanisms or controls 220 for linking the 
content data 210 with the supplemental information and/or action 230 can be 
distributed via one or more communication media 260. Further, they may be obtained 
25 from two or more different sources such as two or more sites on the Internet, or the 
video can be delivered via a cable television network. Other combinations are possible. 
While single-headed arrows are depicted in Fig. 2, it will be reading appreciated from 
the disclosure that communication between the components of Fig. 2 may be bi- 
directional. 



30 
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The time-sequence data 210, link mechanisms 220 and supplemental information 
and/or action 230 are provided to a playback controller and/or interpreter PCI 240, 
The PCI 240 merges the video, animation and/or video data 210 and the supplemental 
information and/or action 230 for presentation on an appropriate presentation device 
("display") 250. Preferably, the PCI240 has a cache for storing linking mechanisms 
220. It also preferably has a cache for storing supplemental information and/or action 
230. Again, the apparatus 200 can be implemented using a general-purpose computer 
or electronically. 

For example, during a video program delivered to the apparatus 200 via a network 
either on demand or on pre-scheduled basis to viewers, sponsors of the program or 
advertisers can buy advertising slots. The advertising slots can be both in terms of 
tune-length as well as the time-position in the video program. The advertisers can 
present their HTML/XML/other-formats advertisements on a viewer's display 250. 
Using a URL, advertisement content can be downloaded. An advertisement can 
provide a Hnk to the advertiser's e-commerce facility, or provide other means of 
conducting business. This gives advertisers full-control over their advertisement 
content, while the advertisement length and position is pre-negotiated and controlled 
by the video service provider. 

Consider a video program that airs at 8:00 PM and finishes at 8:30 PM. For this 
program, advertisers can buy slots that are variable length in time. These slots can be 
displayed at fixed points in time during the broadcast of the program. An advertiser 
for watches might display a watch advertisement while a lead actor in the video 
program is wearing the watch, which the advertiser wants to sell to the viewers. At 
the same time, another advertiser for suits display a clothing advertisement, which 
might be longer or shorter than the watch advertisement. For on-demand type 
scenarios, the start time of the advertisements is with respect to the beginning of the 
program rather than in absolute time. The advertiser can get charged by the service 
provider based on a number of factors, including the length of the advertisement slot. 
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the placement of an advertisement during the program, and whether or not the slot is 
shared with someone. 

An example entry in the linking mechanism database or advertisement control database 
is: 

Start-time: 00:05:00 
End-time: 00:06:00 

Address: http://www.watchesRus, com/advert/ 

During the broadcast, the PCI 240 displays the video information 210 and starts 
interpreting the advertisement control 220. At a specified start time for a particular 
advertisement, the PCI 240 fetches the advertisement content from the specified 
address, which is defined in the advertisement control 220 along with the timing 
information. Alternatively, the embodiment can be practiced using annotation to link 
the data 210 with the supplemental information and/or action 230, The advertisement 
content is displayed usmg the display or presentation device 250 (or the advertisement 
content can also be pre-fetched). The PCI 240 stops the display of the advertisement 
content when its duration has e?q)ired. 

The format of the linking mechanism database or advertisement control database 220 
has a format designed to be easy to produce, modify and maintain. It should also be 
efl&cient for transmission purposes and be extensible. Exemplary fields for the format 
are: 

Period of advert display (firom date - to date, which days) 

Start time - End time (in minutes: seconds) 

Size of advert on display (in inches) 

Location of advert on display (11c, Ire, urc. ulc, center etc) 

Address of advertisement content (URL) 

Since the PCI 240 executes the database of linking mechanisms or advertisement 
control database 220, the PCI 240 is developed hand-in-hand with the format of a 
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linking mechanism or advertisement control. In addition, so that more features can be 
added to the PCI 240, the PCI 240 should be well structured and extensible. 



When a video, animation and/or audio program 210 starts, the address of the link 
5 mechanism database or the advertisement control database 220 is transmitted with the 
initialization information. 

Fig. 3 is a block diagram illustrating a variation 300 of the apparatus 200 shown in Fig. 
2. For purposes of brevity, like features in Figs. 2 and 3 are identified with the same 

1 0 reference numerals. Only differing elements or features are identified with numerals of 
the format 3XX. In particular, video, animation and audio 210 is provided to an on- 
the-fly content analyser 310 forming part of the apparatus 300, The content analyser 
3 10 is connected to the PCI 240, and preferably forms part of the PCI 240. The 
content analyser 310 can be used to determine the linking of the supplemental 

15 information and/or action with the video, animation and/or audio data 210, either 

entirely on its own or in combination with the linking mechanisms 220, A dashed-line 
arrow extends between the communication media 260 and the content analyser 310. 
This arrow indicates that optionally the video, animation and/or audio data 210 can be 
delivered directly to the content analyser 310 without passing through the PCI 240 

20 first. This might be done where the content analyser operates completely 

independently or to reducing data processing via the PCI 240 to the content analyser 
310. 

Fig. 6 is a flow diagram of a method of providing value-added content related to at 
25 least one of animation, audio and video data, in the process of presenting the 

animation, audio, and/or video data to a user in accordance with a second embodiment 
of the invention. Processing commences in step 600. In step 602, animation, audio, 
and/or video data or programming is delivered for presentation to a user. In step 604, 
the value-added content is provided separately fi*om the animation, audio and/or video 
30 data. The value-added content is preferably an advertisement and can include a 
universal resource locator (URL), However, as described above in terms of the 
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supplemental information and/or action, the value-added content can include additional 
functionality, services and information. Alternatively, or in addition, the value-added 
content may include a set of instructions for performing operations on a local 
apparatus or a remote apparatus linked with the local apparatus by a network. The 
5 local apparatus can be a personal computer or the like connected to the Internet, and 
the remote apparatus can be a site on the Internet. 

In step 606, a linking mechanism or structure is provided, separate from the animation, 
audio, and/or video data, which can link the value-added content with the animation, 

10 audio, and/or video data in a controllable and on-the-fly manner prior to presentation 
of the animation, audio, and/or video data and the value-added content. The linking 
mechanism can include a start time and a stop time of the value-added content in 
relation to the animation, audio, and/or video data, as well as an indicator of the 
location of the value-added content. Alternatively, the Imking mechanism can include 

15 an annotation associated with the animation, audio, and/or video data. More 

preferably, the linking mechanism is dependent upon the content the animation, audio, 
and/or video data. 

In step 608, the animation, audio, and/or video data and the value-added content are 
20 presented to the user. The value-added content can be presented separately in a device 
for reproducing the animation, audio and/or video data, such as a television or video 
monitor. Alternatively, it can be presented in or on the animation, audio and video 
data in a device for reproducing the animation, audio and video data. Processing 
terminates in step 610. 

25 

As described in relation to the first embodiment, the process of Fig. 6 can be 
implemented as an apparatus and/or a computer program product. 

Fig. 7 is a flow diagram of a method of distributing advertisements with animation, 
30 audio and/or video data delivered for presentation by way of broadcast or streaming 
over a network in accordance with a third embodiment of the invention. Processing 
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commences in step 700. In step 702, animation, audio and/or video data is delivered 
for presentation. In step 704, an advertisement related to the animation, audio and/or 
video data is provided separately. Preferably, the advertisement is capable of 
connecting a user with a remote location via a network by user interaction with the 

5 advertisement. In step 706, one or more advertisement control links are provided for 
linking separate advertisements with the animation, audio and/or video data. In step 
708, the advertisement is presented to a user during presentation of the animation, 
audio and/or video data dependent upon the advertisement control link. Optionally, 
the method includes a step (not shovm) of vending a product or service by means of 

10 electronic commerce or contacting a call centre dependent upon the advertisement. 
Processing terminates in step 710. 

Figs. 8A to 8E are a detailed flow diagram of a method of displaying advertisements in 
association with a movie or video in accordance with a fourth embodiment of the 
15 invention. The flow diagram illustrates many specific details, however, those skilled in 
the art will appreciate that numerous changes can be made to the process without 
departing firom the scope and spirit of the invention. Likewise, while detailed steps of 
a process involving video or movies is set forth, the process can be readily adapted for 
application in relation to audio and/or animation data in view of the disclosure herein. 

20 

With reference to Fig. 8A, processing commences in step 800. In step 802, an 
identifier (e.g. a title) of a movie or video is read, and the sources of the movie/video 
and the advertisement control database are determined. In step 804, the advertisement 
control database is opened. In step 806, a panel in a graphical computing environment 

25 (e.g. Microsofl: Windows 98) for displaying an advertisement is created, and the 
current list of advertisements is set to null In step 808, a panel is also created for 
displaying the movie or video. The movie preferably plays in this panel as a separate 
thread. After step 808, three separate threads are carried out preferably substantially in 
parallel by the operating system. In Fig. 8A, the separate threads are denoted by 

30 connectors A, B and C corresponding to Figs. 8B. 8C, and 8D, respectively. 
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Fig. 8B is a flow diagram of a thread for handling incoming advertisement controls and 
advertisements. The advertisement controls and advertisements are streamed for 
presentation of the advertisements to a viewer. In step 812, the media time of the 
movie is obtained from the media player. In step 814, a cache storing advertisement 

5 controls is updated. In step 816, the locations of new advertisements are obtained 
from the updated advertisement control data cache. In step 818, the advertisement 
cache is updated by streaming the advertisements for the current portion of the movie 
playing and for an additional predefined period yet to be played In decision block 820, 
a check is made to determine if the advertisement update thread (Fig. 8C) or the movie 

1 0 playing thread has ended. If decision block 820 returns false (NO), processing 

continues at step 812, Otherwise, if decision block 820 returns true (YES), the thread 
terminates in step 822. 

The process of Fig. 8B can be readily implemented for handling the use of annotations 
15 to link the advertisements and the movie. The annotations of the movie can be 

obtained for the currently playing portion of the movie and a few annotations read 
ahead for a portion(s) of the movie yet to be played. The annotations in the movie are 
then matched with the annotations of the advertisements in the advertisement control 
data cache. The locations of the advertisements are determined in this manner. The 
20 advertisements are then streamed to update the advertisement cache. 

Fig. 8C is a flow diagram of the process of presenting advertisements while a movie is 
playing. In particular, the advertisements are updated. In step 826, the media time of 
the player presenting the movie is read. In step 828, using the media time, a start time 

25 and an end time of a current advertisement(s), a check is made to determine if any 
advertisements have to be removed. This step can be readily modified to handle the 
case where annotation is used to link advertisements with the movie. This involves 
comparing the annotation of the movie with the annotation of the advertisement. The 
advertisement is to be played for the duration of the matching movie annotation. 

30 Decision block 830 determines this result. If decision block 830 returns false (NO), 

processing continues at step 834. Otherwise, if decision block 830 returns true (YES), 
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the advertisement being processed is removed from the advertisement panel created in 
step 806. Processing then continues in step 834. 

In step 834, using the media time, advertisements are selected from the advertisement 
5 cache to be made current. In decision block 836, a check is made to determine if any 
advertisements should be added for presentation. If decision block 836 returns false 
(NO), processing continues at decision block 840. Otherwise, if decision block 836 
returns (YES), processing continues at step 838. In step 838, the advertisement is 
added to the advertisement panel created in step 806 for presentation. Processing 
10 continues at decision block 838. In decision block 840, a check is made to determine if 
the movie has ended or if the user has stopped play. If decision block 840 returns false 
(NO), processing continues at step 826. Otherwise, if decision block 840 returns true 
(YES), processing continues at step 842. In step 842, the player software appHcation 
is closed or terminated and the advertisement control database is closed. Processing of 
1 5 this thread terminates in step 844. 

Fig. 8D is a flow diagram of the process of playing a movie. In step 848, user inputs 
including fast forward, rewdnd, etc., are received and responded to by the player. In 
decision block 850, a check is made to determine if the end of the movie has been 
20 reached or it has been user terminated. If decision block 850 returns false (NO), 

processing continues at step 848. Otherwise, if decision block 850 returns true (YES), 
the thread terminates in step 852. 

Fig. 8E is a flow diagram illustrating the processing of interactive user input in relation 
25 to a playing advertisement. Processing commences in step 870. In step 874, input can 
be received from a user. In step 876, the thread responds to the user input and 
performs an appropriate action as defined in the advertisement control database for the 
advertisement. In decision block 878, a check is made to determine if the 
advertisement has been removed from the advertisement panel. If decision block 878 
30 returns false (NO), processing continues at step 874. Otherwise, if decision block 878 
returns true (YES), the thread terminates in step 880. 
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Thus, Figs. 8A-8E provide specific implementation details of one particular application 
of the fourth embodiment in relation to movie or video data. The process can be 
practiced with readily apparent changes in view of the disclosure for processing 

5 animation and/or audio data. This embodiment is able to strongly link advertising with 
the content of animation, audio and/or video data. In particular, it enables flexible 
control of the flow of the advertisement and the advertisment content. Importantly, 
the process of Figs. 8A-8E provides the advertisment for presentation to a user 
without requiring any modification of the movie itself and the delivery mechanism used 

10 to deliver the movie. The fourth embodiment enables customised and focused content- 
linking of advertising to the time-sequence data. 

The foregoing embodiments of the invention disclose a method, apparatus and 
computer program product for linking supplementary information, actions, or both, to 

15 time-sequence data dependent upon the content of the time-sequence data. The time- 
sequence data include animation, audio and/or video data. Only a small number of 
embodiments have been disclosed by way of example. However, those skilled in the 
art will recognise that the invention can be practiced, with modification, in the light of 
the information contained herein without departing firom the scope and spirit of the 

20 invention. 
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The claims defining the invention are as follows: 

1. A method of presenting supplemental information, action, or both, with 
time-sequence data, including: 

providing time-sequence data; 

separately providing supplemental information, action, or both, fi*om said time- 
sequence data; 

linking said supplemental information, action, or both, with said time-sequence 
data during playback of said time-sequence data. 

2. The method according to claim 1, wherein said supplemental 
information, action, or both, is linked with said time-sequence data by a start tune and 
a stop time. 

3. The method according to claim 1, wherein said supplemental 
information, action, or both, is linked with said time-sequence data by annotation 
information associated with said time-sequence data. 

4. The method according to any one of claims 1 to 3, wherein said 
supplemental information, action, or both, is content-linked with said time sequence 
data. 

5. The method according to claim 4, wherein said supplemental 
information, action, or both, is linked with said time-sequence data dependent upon 
on-the-fly analysis of the content of said time-sequence data during playback of said 
time-sequence data. 

6. The method according to claim 1, wherein said time- sequence data is 
animation, audio, and/or video data. 
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7. The method according to claim 1, wherein said time-sequence data is 
provided by broadcast. 



8, The method according to claim 1, wherein said time-sequence data is 
5 provided via a network. 

9. The method according to claim 1, wherein said supplemental 
information, action, or both, is provided remotely via a network. 

10 10. The method according to claim 1, wherein said supplemental 

information, action, or both, includes a displayed prompt with which a user can interact 
by user input. 

11. The method according to claim 1, wherein said supplemental 
15 information, action, or both, includes a link to a remote location. 

12. The method according to claim 1 1, wherein said link is a universal 
resource locator (URL). 

20 13. The method according to claim 1, wherein said time-sequence data and 

said supplemental information, action, or both are linked by a link means. 

14. The method according to claim 13, wherein two or more of said link 
means, said time-sequence data, and said supplemental information, action, or both, are 

25 initially co-located at their source. 

15. The method according to claim 13, wherein two or more of said link 
means, said time-sequence data, and said supplemental information, action, or both, are 
initially remotely located at one or more source locations. 

30 
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16. An apparatus for presenting supplemental information, action, or both, 
with time-sequence data, including: 

means for providing time-sequence data; 

means for separately providing supplemental information, action, or both, from 
5 said time-sequence data; and 

means for linking said supplemental information, action, or both, with said 
time-sequence data during playback of said time-sequence data. 



17. The apparatus according to claim 16, wherein said supplemental 

10 information, action, or both, is linked with said time-sequence data by a start time and 
a stop time. 

18. The apparatus according to claim 16, wherein said supplemental 
information, action, or both, is linked with said time-sequence data by annotation 

1 5 information associated with said time-sequence data. 



19. The apparatus according to any one of claims 16 to 18, wherein said 
supplemental information, action, or both, is content-linked with said time sequence 
data. 

20 

20. The apparatus according to claim 19, wherein said supplemental 
information, action, or both, is linked with said time-sequence data dependent upon 
on-the-fly analysis of the content of said time-sequence data during playback of said 
time-sequence data. 

25 

21 . The apparatus according to claim 16, wherein said time-sequence data 
is animation, audio, and/or video data. 



22. The apparatus according to claim 16, wherein said time-sequence data 
30 is provided by broadcast. 
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23. The apparatus according to claim 16, wherein said time-sequence data 
is provided via a network. 

24. The apparatus according to claim 16, wherein said supplemental 
5 information, action, or both, is provided remotely via a network. 

25. The apparatus according to claim 16, wherein said supplemental 
information, action, or both, includes a displayed prompt with which a user can interact 
by user input. 

10 

26. The apparatus according to claim 16, wherein said supplemental 
information, action, or both, includes a link to a remote location. 



27. The apparatus according to claim 16, wherein said link is a universal 
1 5 resource locator (URL) . 

28. The apparatus according to claim 16, wherein said time-sequence data 
and said supplemental information, action, or both are linked by a link means. 

20 29, The apparatus according to claim 16, wherein two or more of said link 

means, said time-sequence data, and said supplemental information, action, or both, are 
initially co-located at their source. 

29. The apparatus according to claim 28, wherein two or more of said link 
25 means, said time-sequence data, and said supplemental information, action, or both, are 

initially remotely located at one or more source locations. 



30. A computer program product ha\ang a computer readable medium 
having a computer program recorded therein for presenting supplemental information, 
30 action, or both, with time-sequence data, including. 

means for providing time-sequence data; 
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means for separately providing supplemental information, action, or both, from 
said time-sequence data; and 

means for linking said supplemental information, action, or both, with said 
time-sequence data during playback of said time-sequence data. 

5 

3 1 . The computer program product according to claim 30, wherein said 
supplemental information, action, or both, is Unked with said time-sequence data by a 
start time and a stop time. 

10 32. The computer program product according to claim 30, wherein said 

supplemental information, action, or both, is linked with said time-sequence data by 
annotation information associated with said time-sequence data. 

33. The computer program product according to any one of claims 30 to 
15 32, wherein said supplemental information, action, or both, is content-Unked with said 

time sequence data. 

34. The computer program product according to claim 33, wherein said 
supplemental information, action, or both, is linked with said time-sequence data 

20 dependent upon on-the-fly analysis of the content of said time-sequence data during 
playback of said time-sequence data. 

35. The computer program product according to claim 30, wherein said 
time-sequence data is animation, audio, and/or video data. 

25 

36. The computer program product according to claim 30, wherein said 
time-sequence data is provided by broadcast. 



37. The computer program product according to claim 30, wherein said 
30 time-sequence data is provided via a network. 
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38. The computer program product according to claim 30, wherein said 
supplemental information, action, or both, is provided remotely via a network. 



39. The computer program product according to claim 30, wherein said 
5 supplemental information, action, or both, includes a displayed prompt with which a 

user can interact by user input. 

40. The computer program product according to claim 30, wherein said 
supplemental information, action, or both, includes a link to a remote location. 

10 

41. The computer program product according to claim 30, wherein said 
link is a universal resource locator (URL). 

42. The computer program product according to claim 30, wherein said 
15 time-sequence data and said supplemental information, action, or both are linked by a 

link means. 

43. The computer program product according to claim 30, wherein two or 
more of said link means, said time-sequence data, and said supplemental information, 

20 action, or both, are initially co-located at their source. 

44. The computer program product according to claim 43, wherein two or 
more of said link meaas, said time-sequence data, and said supplemental information, 
action, or both, are initially remotely located at one or more source locations. 

25 

45. A method of providing value-added content related to at least one of 
animation, audio, and video data, while said at least one of animation, audio, and video 
data is being presented to a user, including the steps of: 

delivering said at least one of animation, audio, and video data for presentation 
30 to said user; 
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providing said value-added content being separate from said at least one of 
animation, audio and video data; and 

providing linking means, separate from said at least one of animation, audio, 
and video data, for linking said value-added content with said at least one of animation, 
5 audio, and video data in a controllable and on-the-fly manner prior to presentation of 
said at least one of animation, audio, and video data and said value-added content. 

46. The method according to claim 45, ftirther mcluding the step of 
presenting for said user said at least one of animation, audio, and video data and said 

1 0 value-added content. 

47. The method according to claim 45 or 46, wherein said value-added 
content is an advertisement. 

15 48. The method according to claim 45, wherein said linking means includes 

a start time and a stop time of said value-added content in relation to said at least one 
of said animation, audio, and video data. 

49, The method according to claim 45, wherein said linking means includes 
20 an annotation associated with said at least one of said animation, audio, and video 

data. 

50, The method according to claim 45, wherein said value-added content 
includes a universal resource locator (URL). 

25 

5 1 , The method according to claim 45, wherein said value-added content 
includes a set of instructions for performing operations on a local apparatus or a 
remote apparatus linked with said local apparatus by a network. 
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52. The method according to claim 46, wherein said value-added content is 
presented separately in a device for reproducing said at least one of animation^ audio 
and video data. 

5 53. The method according to claim 46, wherein said value-added content is 

presented in or on said at least one of animation, audio and video data in a device for 
reproducing said at least one of animation, audio and \ddeo data. 

54. The method according to claim 45, wherein said linking means is 
10 dependent upon the content said at least one of animation, audio, and video data. 

55. An apparatus for providing value-added content related to at least one 
of animation, audio, and video data, while said at least one of animation, audio, and 
video data is being presented to a user, including: 

means for delivering said at least one of animation, audio, and video data for 
presentation to said user; 

means for providing said value-added content being separate from said at least 
one of animation, audio and video data; and 

means for providing linking means, separate from said at least one of 
animation, audio, and video data, for linking said value-added content with said at least 
one of animation, audio, and video data in a controllable and on-the-fly manner prior to 
presentation of said at least one of animation, audio, and video data and said value- 
added content. 

25 56. The apparatus according to claim 55, further including means for 

presenting to said user said at least one of animation, audio, and video data and said 
value-added content. 

57. The apparatus according to claim 55 or 56, wherein said value-added 
30 content is an advertisement. 
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58. The apparatus according to claim 55, wherein said linking means 
includes a start time and a stop time of said value-added content in relation to said at 
least one of said animation, audio, and video data. 

59. The apparatus according to claim 55, wherein said linking means 
includes an annotation associated with said at least one of said animation, audio, and 
video data. 

60. The apparatus according to claim 55, wherein said value-added content 
includes a universal resource locator (URL). 

6 1 . The apparatus according to claim 55, wherein said value-added content 
includes a set of instructions for performing operations on a local apparatus or a 
remote apparatus linked with said local apparatus by a network. 

62. The apparatus according to claim 56, wherein said value-added content 
is presented separately in said means for representing said at least one of animation, 
audio and video data. 

63. The apparatus according to claim 56, wherein said value-added content 
is presented in or on said at least one of animation, audio and video data in said means 
for reproducing said at least one of animation, audio and video data. 

64. The apparatus according to claim 55, wherein said linking means is 
dependent upon the content said at least one of animation, audio, and video data. 

65. A computer program product having a computer readable medium 
having a computer program recorded therein for providing value-added content related 
to at least one of animation, audio, and video data, while said at least one of animation, 
audio, and video data is being presented to a user, including- 
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means for delivering said at least one of animation, audio, and video data for 
presentation to said user; 

means for providing said value-added content being separate from said at least 
one of animation, audio and video data; and 
5 means for providing linking means, separate from said at least one of 

animation, audio, and video data, for linking said value-added content with said at least 
one of animation, audio, and video data in a controllable and on-the-fly manner prior to 
presentation of said at least one of animation, audio, and video data and said value- 
added content. 

10 

66. The computer program product according to claim 65, further including 
means for presenting to said user said at least one of animation, audio, and video data 
and said value-added content. 

15 67. The computer program product according to claim 65 or 66, wherein 

said value-added content is an advertisement. 

68. The computer program product according to claim 65, wherein said 
linking means includes a start time and a stop time of said value-added content in 

20 relation to said at least one of said animation, audio, and video data 

69. The computer program product according to claim 65, wherein said 
linking means includes an annotation associated with said at least one of said 
animation, audio, and video data. 

25 

70. The computer program product according to claim 65, wherein said 
value-added content includes a universal resource locator (URL). 



71 . The computer program product according to claim 65, wherein said 
30 \^alue-added content includes a set of instructions for performing operations on a local 
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computer program product or a remote computer program product linked with said 
local computer program product by a network. 



72, The computer program product according to claim 66, wherein said 
5 value-added content is presented separately in said means for presenting said at least 

one of animation, audio and video data. 

73. The computer program product according to claim 66, wherein said 
value-added content is presented in or on said at least one of animation, audio and 

10 video data in said means for reproducing said at least one of animation, audio and 
video data. 

75. The computer program product according to claim 65, wherein said 
linking means is dependent upon the content said at least one of animation, audio, and 

15 video data. 

76. A method of distributing advertisements with animation, audio and/or 
video data, said animation, audio and/or video data being delivered for presentation by 
way of broadcast or streaming over a network, said method including the steps of 

20 delivering said animation, audio and/or video data for presentation; 

delivering a separate advertisement related to said animation, audio and/or 
video data; 

providing an advertisement control link for Unking said separate advertisement 
with said animation, audio and/or video data; and 
25 presenting for a user said advertisement during presentation of said animation, 

audio and/or video data dependent upon said advertisement control link. 

77. The method according to claim 76, wherein said advertisement is 
capable of connecting a user with a remote location via a network by user interaction 

30 with said advertisement. 
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78. The method according to claim 11, further comprising the step of 
vending a product or service by means of electronic commerce or contacting a call 
centre dependent upon said advertisement. 

79. An apparatus for distributing advertisements with animation, audio 
and/or video data, said animation, audio and/or video data being delivered for 
presentation by way of broadcast or streaming over a network, said apparatus 
including: 

means for delivering said animation, audio and/or video data for presentation; 
means for delivering a separate advertisement related to said animation, audio 
and/or video data; 

means for providing an advertisement control Unk for linking said separate 
advertisement with said animation, audio and/or video data; and 

means for presenting for a user said advertisement during presentation of said 
animation, audio and/or video data dependent upon said advertisement control Unk. 

80. The apparatus according to claim 79, wherein said advertisement is 
capable of connecting a user with a remote location via a network by user interaction 
with said advertisement. 

8 1 . The apparatus according to claim 80, further comprising means for 
vending a product or service using electronic commerce or by contacting a call centre 
dependent upon said advertisement. 

82. An computer program product for distributing advertisements with 
animation, audio and/or video data, said animation, audio and/or video data being 
delivered for presentation by way of broadcast or streaming over a network, said 
computer program product including: 

means for delivering said animation, audio and/or video data for presentation; 
means for delivering a separate advertisement related to said animation, audio 
and/or video data; 
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means for providing an advertisement control link for linking said separate 
advertisement with said animation, audio and/or video data; and 

means for presenting for a user said advertisement during presentation of said 
animation, audio and/or video data dependent upon said advertisement control link. 

5 

83. The computer program product according to claim 82, wherein said 
advertisement is capable of connecting a user with a remote location via a network by 
user interaction with said advertisement. 

1 0 84. The computer program product according to claim 83, further 

comprising means for vending a product or service using electronic commerce or by 
contacting a call centre dependent upon said advertisement. 
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METHOD AND APPARATUS FOR CONTENT-LINKING SUPPLEMENTAL 
INFORMATION WITH TIME-SEQUENCE DATA 

5 ABSTRACT 

A method, apparatus and computer program product are disclosed for linking 
supplemental information and/or actions (120-124) on-the-fly with animation, audio 
and/or video data (110) for presentation, dependent upon the content of the 
animation, audio and/or video data (110). The supplemental information and/or 

10 actions (120-124) are preferably advertisements and means of electronic commerce 
linked on the fly with the animation, audio and/or video data (110). In this manner, 
advertisements (162-168) can be flexibly and controUably delivered in combination 
with the animation, audio and/or video data (160). The latter may be dehvered by 
broadcasting or by streaming over an electronic network. The animation, audio and/or 

15 video data (210), advertisements (230) and an advertisement control database (220) 
containing linking mechanisms or structures are provided to a playback controller 
and/or interpreter (240), which can be implemented using a personal computer or a set 
topbox, prior to being displayed on a display or other presentation mechanism (250), 
The user can interact with the supplemental infonnation and/or action (250). 

20 
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30086 



as my/our attorney(s) or agent(s) to prosecute the application identified above, and to transact all 
business in the Patent and Trademarl< Office connected therewith. 



Please change the correspondence address for the above-identified application to: 
rn The above-mentioned Customer Number. 
OR 



Firm or 

Individual Name 



Address 



LADAS & PARRY 



26 West 61st Street 



Address 



City 



New York 



State 



NY 



.ZIE. 



10023 



Country 



United States of America 



Telephone 



(212) 708 1800 



Fax 



(212) 246 8959 



I am the: 

Applicant. 



□ 



Assignee of record of the entire interest 
Certificate under 37 CFR 3, 73(b) is enclosed 



SIGNATURE of Applicant or Assignee of Record 



Name 



VASUDEVAN, VInod Vijayalekshmi 



Signature 



T 



Date 
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DECLARATION FOR UTILITY OR 
DESIGN 
PATENT APPLICATION 
(37 CFR1.63) 



El Declaration 
Submitted 
with Initial 
Filing 



□ Declaration 
OR Submitted after Initial 
Filing (surcharge 
(37 CFR 1.16(e)) 
required) 



Attorney Docket Number 



First Named Inventor 



SINGH, Gurminder 



COMPLETE IF KNOWN 



Application Number 



Filing Date 



Group Art Unit 



Examiner Name 



As a below named Inventor, i hereby decEare that: 

My residence, post office address, and citizenship are as stated below next to my name, 

I believe 1 am the original, first and sole inventor {rf only one name Is listed below) or an original, first and joint inventor {if plural 
names are listed below) of the subject matter which is claimed and for which a patent is sought on the invention entitled: 



Method And Apparatus For Content-Linking Supplemental Information 
With Time-Sequence Data 



the specification of which 

^ is attached hereto 
OR 

El was filed on (MM/DD/YYYY) 



(T/tfe of the Invention) 



11/27/1998 



as United States Application Number or PCT International 

I (If applicable). 



Application Number j PGT/SG98/00096 j and was amended on (MM/DD/YYYY) j 

I hereby state that I have reviewed and understand the contents of the above identified specification, including the claims, as 
amended by any amendment specifically referred to above. 

1 acknowledge the duty to disclose infonnation which is material to patentability as defined in 37 CFR 1.56. 



i hereby claim foreign priority benefits under 35 U.S.C. 119(a)-(d) or 365(b) of any foreign application(s) for patent or inventor's 
certificate, or 365(a) of any PCT international application which designated at least one country other than the United States of 
America, listed below and have also identified below, by checking the box, any foreign application for patent or inventor's certificate, 
or of any PCT international application having a filing date before that of the application on which priority Is claimed. 



Prior Foreign Appiication 
Number(s) 



Country 



Foreign Filing Date 
(MM/DD/YYYY) 



Priority 
Not Ciainrted 



Certified Copy Attached? 

Yis m 



□ 
□ 

□ 



□ 
□ 
□ 
□ 



□ 
□ 
□ 
□ 



□ Additional foreign application numbers are listed on a supplemental priority data sheet PTO/SB/02B attached hereto: 



I herebv claim the benefit under 35 U.S.C> 119(e) of any United States provisional application(sl listed below. 



Application Number(s) 



Filing Date (MM/DD/YYYY) 



I I Additional provisional application 
nunnbers are listed on a 
supplemental priority data sheet 
PTO/SB/02B attached hereto. 
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